1. Field of the Invention
The present invention relates to image processing apparatuses and methods, and in particular, to an image processing apparatus and method adapted for, e.g., converting a standard-definition image into a high-definition image.
2. Description of the Related Art
In the case where a standard-definition or low-definition image (hereinafter referred to as an xe2x80x9cSD imagexe2x80x9d) is converted into a high-definition image (hereinafter referred to as an xe2x80x9cHD imagexe2x80x9d), or an image is enlarged, a general xe2x80x9cinterpolation filterxe2x80x9d or the like is used to interpolate (compensate) the levels of missing pixels.
Using the interpolation filter to interpolate the pixels however cannot restore components (high-frequency components) of the HD image, which are not included in the SD image. Thus, a high-definition image cannot be obtained.
Accordingly, the present inventor has already proposed an image converting apparatus for converting an SD image into an HD image including high-frequency components which are not included in the SD image.
The image converting apparatus uses linear coupling of an SD image and prediction coefficients to perform adaptive processing for finding the predicted values of pixels in an HD image, whereby high-frequency components not included in the SD image can be restored.
It is assumed that the predicted value E[y] of pixel level y on a pixel (hereinafter referred to as a xe2x80x9cHD pixelxe2x80x9d) included in the HD image is found using a first-order linear coupling model defined by linear coupling of the levels (hereinafter referred to as xe2x80x9clearning dataxe2x80x9d) x1, x2 . . . of a plurality of SD pixels, and predetermined prediction coefficients w1, w2 . . . In this case, predicted value E[y] can be expressed by the following equation.
E[y]=w1x1+w2x2+ . . . xe2x80x83xe2x80x83(1)
Accordingly, for generalization, when matrix W composed of a set of prediction coefficients w, matrix X composed of a set of learning data, and matrix Yxe2x80x2 composed of a set of predicted values E[y] are defined as follows:       X    =          (                                                  x              11                                                          x              12                                            …                                              x                              1                ⁢                n                                                                                        x              21                                                          x              22                                            …                                              x                              2                ⁢                n                                                                          ⋯                                ⋯                                ⋯                                …                                                              x              m1                                                          x              m2                                            …                                              x              mn                                          )                  W      =              (                                                            W                1                                                                                        W                2                                                                        ⋯                                                                          W                n                                                    )              ,                  Y        xe2x80x2            =              (                                                            E                ⁡                                  [                                      y                    1                                    ]                                                                                                        E                ⁡                                  [                                      y                    2                                    ]                                                                                        ⋯                                                                          E                ⁡                                  [                                      y                    m                                    ]                                                                    )            
the following observation equation holds.
XW=Yxe2x80x2xe2x80x83xe2x80x83(2)
Next, least squares are applied to the observation equation in order to find predicted value E[y] close to pixel level y of the HD image. In this case, when matrix Y composed of a set of true pixel levels y of the HD pixels as teaching data, and matrix E composed of a set of residuals e of predicted values E[y] corresponding to HD pixel levels y are defined as follows:       E    =          (                                                  e              1                                                                          e              2                                                            ⋯                                                              e              m                                          )        ,      Y    =          (                                                  y              1                                                                          y              2                                                            ⋯                                                              y              m                                          )      
the following equation holds from equations (2).
XW=Y+Exe2x80x83xe2x80x83(3)
In this case, prediction coefficient wi for finding E[y] close to HD-image pixel level y can be found by minimizing the following square error:       ∑          i      =      1        m    ⁢      e    i    2  
Therefore, the above square error is differentiated to be zero, in other words, prediction coefficient wi, which satisfies the following equation, is an optimal value for finding predicted value E[y] close to HD-image pixel level y.                                                         e              1                        ⁢                                          ∂                                  e                  1                                                            ∂                                  w                  i                                                              +                                    e              2                        ⁢                                          ∂                                  e                  2                                                            ∂                                  w                  i                                                              +          …          +                                    e              m                        ⁢                                          ∂                                  e                  m                                                            ∂                                  w                  i                                                                    =                  0          ⁢                      (                                          i                =                1                            ,              2              ,              …              ⁢                              xe2x80x83                            ,              n                        )                                              (        4        )            
Initially, by using prediction coefficient wi to differentiate equation (3), the following equations hold.                                                         ∂                              e                1                                                    ∂                              w                i                                              =                      x            i1                          ,                                            ∂                              e                i                                                    ∂                              w                2                                              =                      x            i2                          ,        …        ⁢                  xe2x80x83                ,                                            ∂                              e                i                                                    ∂                              w                n                                              =                      x                          i              ⁢                              xe2x80x83                            ⁢              n                                      ,                  (                                    i              =              1                        ,            2            ,            …            ⁢                          xe2x80x83                        ,            m                    )                                    (        5        )            
From equations (4) and (5), equations (6) are obtained.                                                         ∑                              i                =                1                            m                        ⁢                                          e                i                            ⁢                              x                i1                                              =          0                ,                                            ∑                              i                =                1                            m                        ⁢                                          e                i                            ⁢                              x                                                      xe2x80x83                                    ⁢                  i2                                                              =          0                ,                              …            ⁢                          xe2x80x83                        ⁢                                          ∑                                  i                  =                  1                                m                            ⁢                                                e                  i                                ⁢                                  x                                      i                    ⁢                                          xe2x80x83                                        ⁢                    n                                                                                =          0                                    (        6        )            
In addition, in view of the relationship among learning data x, a set of prediction coefficients w, residuals e, the following normal equations can be obtained from equations (6).                     {                                                                                                                        (                                                                        ∑                                                      i                            =                            1                                                    m                                                ⁢                                                                              x                            i1                                                    ⁢                                                      x                            i1                                                                                              )                                        ⁢                                          w                      1                                                        +                                                            (                                                                        ∑                                                      i                            =                            1                                                    m                                                ⁢                                                                              x                            i1                                                    ⁢                                                      x                            i2                                                                                              )                                        ⁢                                          w                      2                                                        +                  …                  +                                                            (                                                                        ∑                                                      i                            =                            1                                                    m                                                ⁢                                                                              x                            i1                                                    ⁢                                                      x                                                          i                              ⁢                                                              xe2x80x83                                                            ⁢                              n                                                                                                                          )                                        ⁢                                          w                      n                                                                      =                                  (                                                            ∑                                              i                        =                        1                                            m                                        ⁢                                                                  x                        i1                                            ⁢                                              y                        i                                                                              )                                                                                                                                                                    (                                                                        ∑                                                      i                            =                            1                                                    m                                                ⁢                                                                              x                            i2                                                    ⁢                                                      x                            i1                                                                                              )                                        ⁢                                          w                      1                                                        +                                                            (                                                                        ∑                                                      i                            =                            1                                                    m                                                ⁢                                                                              x                            i2                                                    ⁢                                                      x                            i2                                                                                              )                                        ⁢                                          w                      2                                                        +                  …                  +                                                            (                                                                        ∑                                                      i                            =                            1                                                    m                                                ⁢                                                                              x                            i2                                                    ⁢                                                      x                                                          i                              ⁢                                                              xe2x80x83                                                            ⁢                              n                                                                                                                          )                                        ⁢                                          w                      n                                                                      =                                  (                                                            ∑                                              i                        =                        1                                            m                                        ⁢                                                                  x                        i2                                            ⁢                                              y                        i                                                                              )                                                                                        ⋮                                                                                                                                      (                                                                        ∑                                                      i                            =                            1                                                    m                                                ⁢                                                                              x                                                          i                              ⁢                                                              xe2x80x83                                                            ⁢                              n                                                                                ⁢                                                      x                            i1                                                                                              )                                        ⁢                                          w                      1                                                        +                                                            (                                                                        ∑                                                      i                            =                            1                                                    m                                                ⁢                                                                              x                                                          i                              ⁢                                                              xe2x80x83                                                            ⁢                              n                                                                                ⁢                                                      x                            i2                                                                                              )                                        ⁢                                          w                      2                                                        +                  …                  +                                                            (                                                                        ∑                                                      i                            =                            1                                                    m                                                ⁢                                                                              x                                                          i                              ⁢                                                              xe2x80x83                                                            ⁢                              n                                                                                ⁢                                                      x                                                          i                              ⁢                                                              xe2x80x83                                                            ⁢                              n                                                                                                                          )                                        ⁢                                          w                      n                                                                      =                                  (                                                            ∑                                              i                        =                        1                                            m                                        ⁢                                                                  x                                                  i                          ⁢                                                      xe2x80x83                                                    ⁢                          n                                                                    ⁢                                              y                        i                                                                              )                                                                                        (        7        )            
The number of the normal equations as equations (7) that can be formed is identical to the number of sets of prediction coefficients w to be found. Thus, by solving equations (7), sets of optimal coefficients w can be found. (To solve equations (7), it is required in equations (7) that a matrix composed of coefficients as to sets of prediction coefficients w be nonsingular) For solving equations (7), for example, sweeping (Gauss-Jordan elimination) etc. can be applied.
Finding sets of optimal prediction coefficients w in the above manner and using the found sets of prediction coefficients w in equation (1) to find predicted values E[y] close to HD-image pixel levels y is adaptive processing. Adaptive processing differs from interpolation in that components included in the HD image which are not included in the SD image are reproduced. In other words, from only equations (1), it is found that the adaptive processing is identical to interpolation with a general xe2x80x9cinterpolation filterxe2x80x9d. However, since sets of prediction coefficients w corresponding to tap coefficients of the interpolation filter are found by learning with teaching data y, the components included in the HD image can be reproduced. In other words, a high-definition image can be easily obtained. From this fact, it may be said that the adaptive processing has the function of creating an image (definition).
FIG. 12 shows a block diagram of an image converting apparatus for using the above-described adaptive processing to convert an SD image into an HD image.
An SD image is supplied to a classification unit 201 and an adaptive processing unit 204. The classification unit 201 includes a class-tap generating circuit 202 and a classification circuit 203, and classifies HD pixels (HD pixels being referred to)(hereinafter referred to as xe2x80x9creference pixelsxe2x80x9d), as to which predicted values are found, into predetermined classes based on the properties of SD-image pixels corresponding to the reference pixels.
In other words, in the class-tap generating circuit 202, SD pixels corresponding to the reference pixels, which are used to classify the reference pixels (hereinafter referred to as xe2x80x9cclass-tapsxe2x80x9d), for example, a plurality of SD pixels having predetermined positional relationships with respect to the reference pixels are extracted from the SD image supplied to the classification unit 201, and they are supplied to the classification circuit 203. The classification circuit 203 detects the pattern (pixel-level distribution) of SD pixel levels constituting the class-taps from the class-tap generating circuit 202, and supplies a pre-assigned value to the pattern as a reference pixel class to the adaptive processing unit 204.
Specifically, by way of example, it is assumed that the HD image is composed of pixels (HD pixels) indicated by symbols xc3x97 in FIG. 13, and the SD image is composed of pixels (SD pixels) indicated by symbols ◯ in FIG. 13. In FIG. 13, the SD image has half the number of horizontal or vertical pixels as the HD image. In FIG. 13 (similarly to FIG. 14 to FIG. 16 described below), the SD pixel (indicated by ◯ in FIG. 13) at the (i+1)-th position from the left and the (j+1)-th position from the top is denoted by Xi,j, and similarly, the HD pixel (indicated by xc3x97 in FIG. 13) at the (ixe2x80x2+1)-th position from the left and the (jxe2x80x2+1)-th position from the top is denoted by Yi,j. In this case, the position of SD pixel Xi,j and the position of HD pixel Y2i,2j are the same.
Here, if it is assumed that one SD pixel, for example, HD pixel Y4,4 positioned identically to X2,2 is a reference pixel, the class-tap generating circuit 202 extracts and uses as the class tap of the reference pixel (HD pixel) Y4,4, for example, 3xc3x973 (horizontalxc3x97vertical) SD pixels X1,1, X2,1, X3,1, X1,2, X2,2xe2x80x2, X3,2xe2x80x2, X1,3xe2x80x2, X2,3xe2x80x2, and X3,3 (SD pixels in the area defined by the dotted line in FIG. 13) which have SD pixel X2,2 as the center, having possible correlation with HD pixel Y4,4 positioned identically to HD pixel Y4,4.
In the case where, for example, HD pixel Y5,4 on the right of HD pixel Y4,4 positioned identically to X2,2 is used as a reference pixel, the class-tap generating circuit 202 extracts SD pixels corresponding to HD pixel Y5,4, for example, nine SD pixels, surrounded by the dotted line in FIG. 14, including SD pixel X4,2 in place of SD pixel X1,2 in the class tap formed when HD pixel Y4,4 is used as a reference pixel, and the nine SD pixels are used as the class tap of the reference pixel (HD pixel) Y5,4.
In the case where, for example, HD pixel Y4,5 adjacent to HD pixel Y4,4 positioned identically to X2,2 is used as a reference pixel, the class-tap generating circuit 202 extracts SD pixels corresponding to HD pixel Y4,5, for example, nine SD pixels, surrounded by the dotted line in FIG. 15, including SD pixel X2,4 in place of SD pixel X2,1 in the class tap formed when HD pixel Y4,4 is used as a reference pixel, and the nine SD pixels are used as the class tap of reference pixel (HD pixel) Y4,5.
In the case where, for example, HD pixel Y5,5 adjacent below to obliquely right HD pixel Y4,4 is used as a reference pixel, the class-tap generating circuit 202 extracts SD pixels corresponding to HD pixel Y5,5, for example, nine SD pixels, surround by the dotted line in FIG. 16, including SD pixel X4,4 in place of SD pixel X1,1 in the class tap formed when HD pixel Y4,4 is used as a reference pixel, and the nine SD pixels are used as the class tap of the reference pixel (HD pixel) Y5,5.
The classification circuit 203 detects the pattern of the nine SD pixels (pixel levels) as a class tap, and outputs a value corresponding to the pattern as a reference pixel class.
This class is supplied to the address terminal (AD) of a coefficient read only memory (ROM) 207 in the adaptive processing unit 204.
In general, eight bits or the like are assigned to one pixel included in an image. If eight bits are assigned to one SD pixel, the number of pixel level patterns in only the square class tap composed of 3xc3x973 pixels shown in FIG. 13 is as extremely large as (28)9, so that rapid post-processing cannot be performed.
Accordingly, before classification is performed, the pre-processing of the class tap is performed using, for example, adaptive dynamic range coding (ADRC) for reducing the number of bits of SD pixels constituting the class tap.
According to ADRC, a pixel having a maximum level (hereinafter referred to as a xe2x80x9cmaximum pixelxe2x80x9d) and a pixel having a minimum level (hereinafter referred to as a xe2x80x9cminimum pixelxe2x80x9d) are detected from the nine SD pixels constituting the class tap. The difference DR (=MAXxe2x88x92MIN) between the maximum pixel level MAX and the minimum pixel level MIN is computed, and the DR is used as a local dynamic range of the class tap. Based on the dynamic range DR, each pixel level of the class tap is requantized to K bits, which is less than the number of the originally assigned bits. In other words, the minimum pixel level MIN is subtracted from the respective pixel levels of the class tap, and the subtracted values are divided by DR/2K.
As a result, each pixel level of the class tap can be expressed by K bits. Accordingly, when K=1, the number of patterns of the nine SD pixel levels is (21)9. Compared with the case where ADRC is not performed, the number of the patterns can be extremely reduced.
In addition, the adaptive processing unit 204 includes a prediction-tap generating circuit 205, a prediction computation circuit 206, and the coefficient ROM 207, and performs adaptive processing.
The predicted-tap generating circuit 205 extracts from an SD image supplied to the adaptive processing unit 204 a plurality of SD pixels having a predetermined positional relationship with respect to a reference pixel, which are used in the prediction computation circuit 206 to find a predicted value of the reference pixel, and supplies them as a prediction tap to the prediction computation circuit 206.
Specifically, in the case where, HD pixel Y4,4 is used as a reference pixel, and a class tap as described using FIG. 13 is formed, the prediction tap generation circuit 205 extracts SD pixels that may have high correlation with, e.g., HD pixel Y4,4, for example, 5xc3x975 SD pixels (in the area all defined by the solid line in FIG. 13) having SD pixel X2,2 as the center, which is positioned identically to the reference pixel Y4,4, and the SD pixels are used as the prediction tap of the reference pixel (HD pixel) Y4,4.
In the case where, for example, HD pixel Y5,4 is used as a reference pixel, the prediction-tap generating circuit 205 extracts 25 SD pixels surrounded by the solid line in FIG. 14 which include SD pixel X5,2 in place of SD pixel X0,2 in the prediction tap formed when HD pixel Y4,4 is used as a reference pixel, and the 25 SD pixels are used as the class tap of the reference pixel (HD pixel) Y5,4.
In the case where, for example, HD pixel Y4,5 is used as a reference pixel, the prediction-tap generating circuit 205 extracts 25 SD pixels surrounded by the solid line in FIG. 15 which include SD pixel X2,5 in place of SD pixel X2,0 in the prediction tap formed when HD pixel Y4,4 is used as a reference pixel, and the 25 SD pixels are used as the class tap of the reference pixel (HD pixel) Y4,5.
In the case where, for example, HD pixel Y5,5 is used as a reference pixel, the prediction-tap generating circuit 205 extracts 25 SD pixels surrounded by the solid line in FIG. 16 which include SD pixel X5,5 in place of SD pixel X0,0 in the class tap formed when HD pixel Y4,4 is used as a reference pixel, and the 25 SD pixels are used as the class tap of the reference pixel (HD pixel) Y5,5.
In addition, the prediction taps from the prediction-tap generating circuit 205, and the sets of prediction coefficients from the coefficient ROM 207 are supplied to the prediction computation circuit 206.
The coefficient ROM 207 holds the sets of prediction coefficients found by performing learning beforehand for classes. When being supplied with a class from the classification circuit 203, it reads a prediction coefficient stored in the address corresponding to the class, and supplies the read coefficient to the prediction computation circuit 206.
In this manner, the prediction computation circuit 206 is supplied with the prediction tap corresponding to a reference pixel and a set of prediction coefficients as to the reference pixel class. The prediction computation circuit 206 implements the computation shown in equation (1) using prediction coefficients w, w2, . . . from the coefficient ROM 207 and (SD pixels constituting) prediction taps x1, x2, . . . from the prediction-tap generating circuit 6, whereby the predicted value E[y] of the reference pixel (HD pixel) is found, and it is output as an HD pixel level.
The foregoing processing is performed with all HD pixels used as reference pixels, whereby an SD image is converted into an HD image. The class-tap generating circuit 202 and the prediction-tap generating circuit 205 use an identical HD pixel in processing.
FIG. 17 shows a block diagram of a learning apparatus that performs learning for computing sets of prediction coefficients for classes which are stored in the coefficient ROM 207 in FIG. 12.
An HD image to be teaching data y in learning is supplied to a decimation circuit 211 and a teaching data extracting circuit 146. In the decimation circuit 211, the HD image is reduced by decimating the number of its pixels, whereby an SD image is formed. In other words, the decimation circuit 211 reduces the numbers of horizontal and vertical pixels of the HD image to form the SD image. The SD image is supplied to a classification unit 212 and a prediction-tap generating circuit 145.
The classification unit 212 or the prediction-tap generating circuit 145 performs processing identical to that in the classification unit 201 or the prediction-tap generating circuit 205, thereby outputting the class or prediction tap of the reference pixel. The class output from the classification unit 212 is supplied to the address terminals (AD) of a prediction tap memory 147 and a data memory 148, and the prediction tap output from the prediction-tap generating circuit 145 is supplied to the prediction tap memory 147. The classification unit 212 and the prediction-tap generating circuit 145 use an identical HD pixel as a reference pixel.
In the prediction tap memory 147, the prediction tap supplied from the prediction-tap generating circuit 145 is stored at the address corresponding to the class supplied from the classification unit 212.
In addition, the teaching data extracting circuit 146 extracts HD pixels that are used as reference pixels in the prediction-tap generating circuit 145 from the supplied HD image, and supplies them as teaching data to the teaching data memory 148.
In the teaching data memory 148, the teaching data supplied from the teaching data extracting circuit 146 are stored at the address corresponding to the class supplied from the classification circuit 146.
The foregoing processing is performed while the HD pixels of all HD images prepared beforehand for learning are being successively used as reference pixels.
As a result, at an identical address of the teaching data memory 148 or the prediction tap memory 14, HD pixels in the class corresponding to the address, or SD pixels positioned so that the prediction tap described in FIGS. 13 to 16 are stored as teaching data y or learning data x.
The prediction tap memory 147 and the teaching data memory 148 hold a plurality of information data at one identical address, whereby, at the identical address, a plurality of learning data x and teaching data y which are classified into one identical class can be stored.
Subsequently, the arithmetic circuit 149 reads from the prediction tap memory 147 or the teaching data memory 148, HD pixels as a prediction tap or a teaching data, stored in the identical address, and uses them to compute sets of prediction coefficients for minimizing the error between the predicted data and the teaching data by least squares. In other words, in the arithmetic circuit 149, the normalization equations (7) hold for each class. By solving these, sets of prediction coefficients are found.
In the above manner, the sets of prediction coefficients for each class, found by the arithmetic circuit 149, are stored at an address corresponding to each class in the coefficient ROM 207 in FIG. 12.
In the above-described learning, there may a case where a class in which normalization equations needed for sets of prediction coefficients are not obtained. Concerning such a class, for example, sets of prediction coefficients etc. obtained by establishing normalization equations, regardless of classes, are used as sets of default prediction coefficients.
According to adaptive processing, an HD image including high-frequency components which are not included in the original SD image can be obtained. By performing classification for reference pixels, and using prediction coefficients in accordance with the obtained classes to perform adaptive processing, adaptive processing suitable for the reference pixels can be performed.
In addition, in the above-described image processing apparatus (FIG. 12) and learning apparatus (FIG. 17), a class tap consists of SD pixels having a positional Ian relationship (described in FIGS. 13 to 16) with respect to a reference pixel, regardless of image characteristics.
In other words, the class tap consists of nine SD pixels close to the reference pixel as described in FIGS. 13 to 16.
In addition, an image can be divided into regions composed of pixels having approximate brightness levels, colors, and so forth: namely, a region in which a predetermined object is displayed, and regions having predetermined colors and patterns. Accordingly, by performing the above-described classification for a region different from others in the image, there may be a case in which all reference pixels in, e.g., a portion having almost no pixel change are classified into one identical class.
However, by performing classification for such a portion so that, for example, SD pixels away from the reference pixels are included in the class tap, reference pixels classified into an identical class may be classified into different classes. In other words, by forming a class tap with SD pixels in a broader range, the reference pixels may be classified into classes adapted for the reference pixels.
If reference pixels can be classified into classes suitable for the reference pixels as described above, adaptive processing more suitable for the reference pixels can be performed, and the quality of the resultant HD image can be increased.
Accordingly, it is an object of the present invention to provide an image processing apparatus and method that enables image quality improvement by solving the above-described problems.
To this end, according to an aspect of the present invention, the foregoing object has been achieved through provision of an image processing apparatus for performing processing for finding a first image by using a second image having a smaller number of pixels than the first image and a third image having a smaller number of pixels than the second image, the image processing apparatus including: a receiving unit for receiving the first image; and a classification unit for classifying reference pixels, which are pixels of the first image being referred to, into predetermined classes in accordance with properties of pixels of the second or third image corresponding to the reference pixels.
According to another aspect of the present invention, the foregoing object has been achieved through provision of an image processing method for performing processing for finding a first image by using a second image having a smaller number of pixels than the first image and a third image having a smaller number of pixels than the second image, the image processing method including the steps of: receiving the first image; and classifying reference pixels, which are pixels of the first image being referred to, into predetermined classes in accordance with properties of pixels of the second or third image corresponding to the reference pixels.
According to an image processing apparatus and image processing method of the present invention, reference pixels as pixels of a first image being referred to are classified into predetermined classes in accordance with properties of second or third image pixels corresponding to the reference pixels. Thus, the reference pixels can be classified into more suitable classes.